Add wrap_entries processor to mutate-event-processors by nishantKadivar · Pull Request #6665 · opensearch-project/data-prepper

nishantKadivar · 2026-03-22T09:31:13Z

Add `wrap_entries` processor to mutate-event-processors

Description

Adds a new wrap_entries processor that wraps each element of a primitive array into an object using a configured key name. This enables downstream processors like add_entries and delete_entries with iterate_on, which require List<Map<String, Object>> and cannot operate on primitive arrays (strings, numbers, booleans).

For example, ["alice", "bob"] becomes [{"name": "alice"}, {"name": "bob"}].

Issues Resolved

#6611

Configuration Options

Option	Required	Type	Default	Description
`source`	Yes	String	—	Key of the primitive array to transform (JSON Pointer)
`target`	No	String	`source`	Key to write the result to (defaults to in-place)
`key`	Yes	String	—	Key name in each resulting object
`exclude_null_empty_values`	No	Boolean	`false`	Filter out null/empty elements before wrapping
`append_if_target_exists`	No	Boolean	`false`	Append to existing target array instead of overwriting
`map_entries_when`	No	String	`null`	Conditional expression evaluated at root event level
`tags_on_failure`	No	List	`null`	Tags to add on processing failure

Check List

New functionality includes testing
New functionality has been documented
Commits are signed per the DCO using --signoff

End-to-End Test Results

Tested manually using Data Prepper's file source and file sink.

Pipeline Configuration

map-entries-pipeline:
  source:
    file:
      path: "pipelines/input.json"
      format: "json"
      record_type: "event"
  processor:
    - wrap_entries:
        source: "/names"
        key: "name"
        map_entries_when: '/type == "users"'
    - wrap_entries:
        source: "/tags"
        key: "value"
        map_entries_when: '/type == "tagged"'
    - wrap_entries:
        source: "/items"
        target: "/inventory_items"
        key: "product"
        map_entries_when: '/type == "inventory"'
    - wrap_entries:
        source: "/scores"
        key: "score"
        map_entries_when: '/type == "grades"'
  sink:
    - file:
        path: "pipelines/output.json"

Input (5 records)

{"names": ["alice", "bob", "charlie"], "type": "users"}
{"tags": ["critical", "network", "security"], "source": "firewall", "type": "tagged"}
{"list_a": [{"url": "http://api.example.com/v1"}, {"url": "http://api.example.com/v2"}], "list_b": [{"url": "http://cdn.example.com/assets"}], "type": "urls"}
{"items": ["laptop", "monitor", "keyboard", "mouse"], "department": "IT", "type": "inventory"}
{"scores": [95, 82, 78, 91, 88], "subject": "math", "type": "grades"}

Output (actual results)

{"names":[{"name":"alice"},{"name":"bob"},{"name":"charlie"}],"type":"users"}
{"tags":[{"value":"critical"},{"value":"network"},{"value":"security"}],"source":"firewall","type":"tagged"}
{"list_a":[{"url":"http://api.example.com/v1"},{"url":"http://api.example.com/v2"}],"list_b":[{"url":"http://cdn.example.com/assets"}],"type":"urls"}
{"items":["laptop","monitor","keyboard","mouse"],"department":"IT","type":"inventory","inventory_items":[{"product":"laptop"},{"product":"monitor"},{"product":"keyboard"},{"product":"mouse"}]}
{"scores":[{"score":95},{"score":82},{"score":78},{"score":91},{"score":88}],"subject":"math","type":"grades"}

E2E Scenarios

#	Scenario	Result
1	String array in-place wrap (`type=users`)	PASS
2	String array wrap with different key (`type=tagged`)	PASS
3	Condition not met — no mutation (`type=urls`)	PASS
4	Separate target preserves original (`type=inventory`)	PASS
5	Integer array wrap (`type=grades`)	PASS

Adds a new map_entries processor that wraps each element of a primitive array into an object using a configured key name. This enables downstream processors like add_entries and delete_entries with iterate_on, which require List of Map and cannot operate on primitive arrays. Example: ["alice", "bob"] -> [{"name": "alice"}, {"name": "bob"}] Configuration options: - source (required): key of the primitive array to transform - target (optional): key to write result to (defaults to source) - key (required): key name in each resulting object - exclude_null_empty_values: filter out null/empty elements - append_if_target_exists: append to existing target array - map_entries_when: conditional expression - tags_on_failure: tags on processing failure Includes 21 unit tests and 3 config tests. Signed-off-by: Nishant Kadivar <nimahesx@amazon.com>

bagmarnikhil · 2026-03-26T21:47:22Z

+
+        if (!event.containsKey(source)) {
+            LOG.warn(EVENT, "Source key [{}] does not exist in event [{}], skipping.", source, event);
+            addTagsOnFailure(event);


Is a missing source key really a failure? In a pipeline processing mixed event types, lots of events won't have this key — tagging all of them as failed seems noisy. The other mutate-event processors (SelectEntriesProcessor, FilterListProcessor) treat missing keys as a silent no-op. I'd drop the addTagsOnFailure here and make it a debug-level log instead, reserving failure tags for actual errors like "source exists but isn't a list."

bagmarnikhil · 2026-03-26T21:48:38Z

+                    continue;
+                }
+            }
+            final Map<String, Object> wrapped = new HashMap<>(1);


Since this map is never modified after the put, Collections.singletonMap(key, element) would be a better fit — immutable, less memory, and communicates that it's a single-entry map.

bagmarnikhil · 2026-03-26T21:49:37Z

+                            "for valid expression syntax", config.getMapEntriesWhen()));
+        }
+
+        if (config.getTagsOnFailure() != null) {


Nit: none of the other mutate-event processors validate tag contents like this (AddEntryProcessor, SelectEntriesProcessor, etc.). Not wrong, but it's inconsistent

To be consistent with the other processor, I have removed tag content validation

bagmarnikhil · 2026-03-26T21:55:39Z

+            result.add(wrapped);
+        }
+
+        if (result.isEmpty()) {


When exclude_null_empty_values: true and all elements get filtered out, this early return leaves the original primitive array untouched. But if even one element
survives, the target gets overwritten with an object array. This means the same processor produces different output types depending on the data:

Input [null, ""] → output stays [null, ""] (primitives)

Input [null, "ok"] → output becomes [{"key": "ok"}] (objects)

Downstream processors like add_entries with iterate_on expect a consistent List — they'll break on the primitive case. I'd remove this early return and let it fall through to event.put(effectiveTarget, result) so the output is always [] when nothing survives filtering.

I have removed the if (result.isEmpty()) { return; } check and let it fall through to event.put(effectiveTarget, result). That way the output is always List<Map<String, Object>> — either [] or [{"key": "val"}, ...] — regardless of whether filtering removed everything.

…ty filtered results - Change missing source key from warn+tags_on_failure to silent debug log, consistent with other mutate-event processors - Remove early return when all elements are filtered out by exclude_null_empty_values, ensuring consistent List<Map> output type - Update test and README to reflect new behavior Signed-off-by: Nishant Kadivar <nimahesx@amazon.com>

kkondaka · 2026-03-29T00:30:23Z

Looks good. Not sure about the name of the processor. Looks like it's doing list to list with each list item becoming a map?

github-actions · 2026-03-30T13:49:12Z

✅ License Header Check Passed

All newly added files have proper license headers. Great work! 🎉

Signed-off-by: Nishant Kadivar <nimahesx@amazon.com>

nishantKadivar · 2026-04-08T14:46:36Z

@kkondaka - The name map_entries was chosen because the processor maps each entry in the list into a key-value structure, aligning with the naming of other mutate-event processors.
Open for suggestion

nishantKadivar · 2026-04-09T15:30:34Z

The following newly added files are missing required license headers:

data-prepper-plugins/mutate-event-processors/src/test/resources/org/opensearch/dataprepper/plugins/processor/mutateevent/map_entries_when_filters_records.yaml
Please add the appropriate license header to each file and push your changes.

See the license header requirements: https://github.com/opensearch-project/data-prepper/blob/main/CONTRIBUTING.md#license-headers

Regarding this violation, We do not have headers specifically defined in these YAML files, as we observed that they were not included in any of the existing processor files. This was done to maintain consistency across processors. Let me know if that needed to be included.

dlvenable · 2026-04-09T17:12:07Z

Looks good. Not sure about the name of the processor. Looks like it's doing list to list with each list item becoming a map?

I think this is a good point. "Map entries" sounds like it is running a map function. The word map can be confused with a verb.

dlvenable

Thank you @nishantKadivar for this contribution. I have a few comments. We should change the name as well.

dlvenable · 2026-04-09T17:09:05Z

+        return new Record<>(JacksonEvent.builder().withEventType("event").withData(data).build());
+    }
+
+    // --- Constructor validation delegation ---


Let's avoid these comments for sections. Remove them or use @Nested classes.

dlvenable · 2026-04-09T17:09:16Z

@@ -0,0 +1,10 @@
+test-pipeline:


This also needs a header.

dlvenable · 2026-04-09T17:17:56Z

+                    continue;
+                }
+            }
+            result.add(Collections.singletonMap(key, element));


I suspect that JacksonEvent will take this as-is and not remap. The singletonMap is immutable and thus would fail to mutate in downstream processors. Let's create a new map and put that.

dlvenable · 2026-04-09T17:21:17Z

Looks good. Not sure about the name of the processor. Looks like it's doing list to list with each list item becoming a map?

I think this is a good point. "Map entries" sounds like it is running a map function. The word map can be confused with a verb.

The key verb you use in the README is "wrap." This processor name should start with wrap_ since that is the action it is taking.

dlvenable · 2026-04-09T17:26:03Z

+
+    @Test
+    void doExecute_with_string_array_wraps_each_element_into_object_in_place() {
+        final Record<Event> record = createEvent(Map.of("names", Arrays.asList("alpha", "beta")));


Let's add some test cases when the names are not primitives.

e.g.

List.of(Map.of("name": "alpha"), Map.of("name": "beta"))

and

List.of(List.of("alpha1": "beta1"), List.of("alpha2": "beta2"))

I have added test cases for both the scenarios.

Signed-off-by: Nishant Kadivar <nimahesx@amazon.com>

dlvenable · 2026-04-10T13:46:19Z

    }
+
+    @Test
+    void doExecute_with_map_elements_wraps_each_map_into_object() {


Is this the desirable behavior? I'm ok to approve now, but we should follow up and discuss if we want to have this before we release 2.16.

Hello @dlvenable ,
I have raised an issue for the discussion about the wrap_entries behaviour #6753.

nishantKadivar force-pushed the main branch from ca678b8 to 8c5bba7 Compare March 22, 2026 10:04

nishantKadivar marked this pull request as ready for review March 25, 2026 17:40

nishantKadivar requested review from KarstenSchnitter, dinujoh, dlvenable, engechas, graytaylor0, kkondaka, oeyh, san81, sb2k16 and srikanthjg as code owners March 25, 2026 17:40

bagmarnikhil reviewed Mar 26, 2026

View reviewed changes

nishantKadivar force-pushed the main branch from 21e9bd2 to cdb8014 Compare March 27, 2026 04:29

kkondaka mentioned this pull request Mar 29, 2026

Add filter_list processor to filter list elements #6659

Merged

nishantKadivar added 2 commits March 31, 2026 19:22

Merge branch 'opensearch-project:main' into main

d5b5c7b

Refactor map_entries validation into config and add integration test

a39ddea

Signed-off-by: Nishant Kadivar <nimahesx@amazon.com>

Merge branch 'opensearch-project:main' into main

598f3d9

dlvenable requested changes Apr 9, 2026

View reviewed changes

dlvenable reviewed Apr 9, 2026

View reviewed changes

nishantKadivar added 2 commits April 10, 2026 11:45

Merge branch 'opensearch-project:main' into main

99c3f7d

Add chained wrap_entries processor test for nested listsgit

486a302

Signed-off-by: Nishant Kadivar <nimahesx@amazon.com>

dlvenable approved these changes Apr 10, 2026

View reviewed changes

nishantKadivar changed the title ~~Add map_entries processor to mutate-event-processors~~ Add wrap_entries processor to mutate-event-processors Apr 11, 2026

kkondaka approved these changes Apr 14, 2026

View reviewed changes

kkondaka merged commit 79455e6 into opensearch-project:main Apr 14, 2026
73 of 76 checks passed

Conversation

nishantKadivar commented Mar 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Add wrap_entries processor to mutate-event-processors

Description

Issues Resolved

Configuration Options

Check List

End-to-End Test Results

Pipeline Configuration

Input (5 records)

Output (actual results)

E2E Scenarios

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nishantKadivar Mar 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kkondaka commented Mar 29, 2026

Uh oh!

github-actions Bot commented Mar 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ License Header Check Passed

Uh oh!

nishantKadivar commented Apr 8, 2026

Uh oh!

nishantKadivar commented Apr 9, 2026

Uh oh!

dlvenable commented Apr 9, 2026

Uh oh!

dlvenable left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dlvenable commented Apr 9, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

nishantKadivar commented Mar 22, 2026 •

edited

Loading

Add `wrap_entries` processor to mutate-event-processors

nishantKadivar Mar 27, 2026 •

edited

Loading

github-actions Bot commented Mar 30, 2026 •

edited

Loading